Forensic verification utilizing forensic markings inside halftones

ABSTRACT

A forensic verification system extracts a print signature via a print signature extractor from an interior of a halftone contained in an image. The system utilizes a comparator to compare the print signature to a reference signature stored in a registry to determine differences between the print signature and the reference signature. The system utilizes a forensic analyzer to perform a forensic analysis on the signatures based on the comparison to authenticate the image.

CROSS-REFERENCE

This Application is a Continuation of U.S. patent application Ser. No.14/347,370, filed Mar. 26, 2014, entitled FORENSIC VERIFICATIONUTILIZING FORENSIC MARKINGS INSIDE HALFTONES, which claims benefit toPCT Application No. PCT/US2012/024450, filed Feb. 9, 2012, entitledFORENSIC VERIFICATION UTILIZING FORENSIC MARKINGS INSIDE HALFTONES, bothof which are incorporated herein by reference.

BACKGROUND

In environments where sensitive or confidential documents are handled,such as in financial institutions, it is often desirable to have theability to trace who printed a document as well as when and where it wasprinted. For example, it may be desirable to know if a check or otherfinancial instrument was printed from an authorized source (e.g.,printer, location, and so forth) as opposed to being a fraudulent copyof the respective instrument. Electronic means such as bar coding havebeen applied to documents in an attempt to authenticate the legitimacyof the documents. Bar coding, for example, can unacceptably alter theappearance of the documents by embedding superfluous information such asthe bar code image into the document.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example system for performing forensicverification of printed documents utilizing forensic markings insidehalftones.

FIG. 2 illustrates an example of a forensic verification encoding systemfor generating reference signatures from halftone images.

FIG. 3 illustrates an example set of images that represent halftoneencodings that employ forensic markings inside halftones.

FIG. 4 illustrates an example of bandpass filtering of halftones.

FIG. 5 illustrates an example of difference processing for halftones.

FIG. 6 illustrates an example of equi-spaced segmentation of halftones.

FIG. 7 illustrates an example of segmentation of halftones that areanalyzed along halftone cell boundaries.

FIG. 8 illustrates an example of concentric cell and shifted concentriccell analysis for halftones.

FIG. 9 illustrates an example of decoding and forensic verificationsystem utilizing forensic markings inside halftones.

FIG. 10 illustrates an example method for performing forensicverification of printed documents utilizing forensic markings insidehalftones.

FIG. 11 illustrates an example of a forensic analysis system.

DETAILED DESCRIPTION

FIG. 1 illustrates an example system for performing forensicverification of printed documents utilizing forensic markings insidehalftones to generate and verify print signatures. The system 100includes a forensic encoding system 110 (also referred to as theencoding system) that utilizes information inside halftones to encodeforensic information. As shown, the forensic encoding system 110receives a graphic image for encoding forensic information and payload.Output from the encoding system 110 includes a secure hardcopy andregistry and support data that are employed by a forensic recovery andverification system 120 to perform forensic verification of printeddocuments and to recover payload data that had been previously encoded.As used herein, forensic analysis and verification provide the means toauthenticate a printed document. This includes the ability to provestatistically whether or not a document was printed from an authorizedsource or is an unauthorized copy.

As an example of analysis and verification, electronic tickets andvouchers may be distributed from a ticket agency having an agencyprinter generate the tickets for their respective customers. The ticketsprinted from the agency printer are considered authorized and valid uponredemption. If someone were to print additional originals (electroniccopies) or copy the ticket utilizing a scanner and subsequent printer,such printing would be unauthorized and fraudulent. The system 100provides encoding and decoding of covert information placed withinportions of printed documents to enable authentication. “Covert” impliesthat the authentication information is encoded as part of the printedimage and thus undetectable by the naked eye. As can be appreciated, theauthentication processes described herein can be applied to any type ofprinted document where it is desirable to verify that a given documentcorresponds to a unique physical instantiation. This functionality couldalso be used for applications that include but are not limited to caseswhere it is desirable to trace to the origins of the equipment,organization, or people, that generated the document (e.g., provewhether or not the document was generated by a particular printer, by anauthorized printer, or conversely whether document was printed/copiedfrom an unauthorized printing source).

The system 100 can also be employed to authenticate the association oftwo or more items, for example (e.g., a label or medallion with aserialized document on which the label/medallion is affixed). Since grayscale printing processes are inherently bi-tonal, halftoning processescan be utilized for continuous-tone graphics or images by producing thevisual illusion of continuous tone though the arrangement of black andwhite pixels. One type of halftoning can be based on a clustered-dottechnique, where gray levels are rendered with arrays of black and whiteclusters of pixels where the clusters are of varying size and shape. Astegatone is a halftone example that utilizes shifted dot clusters toencode information into a portion of the printed image.

As shown, the encoding system 110 can embed a payload into a halftone ofa graphic image. The payload represents the data to be embedded in thehalftone that can also be later employed as part of the forensicverification process (e.g., to locate a reference print signature in aregistry). The encoding process includes shifting of the dot clusterswithin the halftone to encode the payload within the halftone. Thehalftone can represent any portion of a printed document. For example,this could include a graphical feature such as a circle or square orirregular shaped image object such as waves crashing on the shore, forexample. Although stegatones which include a halftone pattern that holdssteganographic information may be applied as part of the encoding anddecoding process of the system 100, any form of halftone (e.g., with orwithout addition of the payload) can be utilized for the interiorhalftone authentication processes described herein. In one example,halftones may not change from print-to-print and in other examples eachversion of a halftone in each single printed page can be different. Forinstance, a halftone can be generated that varies only with the printingdevice, document author(s), user requesting the print, timestamp,security clearance, network state, or some combination of these and/orpossibly other factors, for example.

The encoding system 110 creates and stores a reference signature in aregistry (described below with respect to FIG. 2) that acts as atraceable fingerprint for a printed document that enables detection ofauthorized or unauthorized printing. The reference signature can bederived from the interior of a printed object or component within theprinted document. For example, if scenic image were to have an encodedpayload, the reference signature can be generated as a function of theinterior dot clusters of the halftone as will be illustrated below withrespect to FIG. 3. In some examples, only the halftone may be employedfor the encoding and decoding processes described herein. In otherexamples, a unique payload may be generated for the same common graphicimage and subsequently used to index the print signature in the registry(e.g., one-to-one mapping between payload and print signatures). In yetanother case, the same payload may be applied to the graphic image manytimes and used to index the resulting print signatures in the registry(e.g., one-to-many mapping between payload and print signatures).

To authenticate a printed document, a captured image to be analyzed canbe processed by the forensic recovery and verification system 120. Thecaptured image (e.g., snapshot image of printed document) should be ofsufficient resolution to enable the halftone interior analysistechniques described herein. For example, a high resolution camera orscanner (e.g., capable of acquiring an image at a resolution of 7200dots per inch (DPI)) can be employed to capture printed media andgenerate the image to be analyzed. The forensic recovery andverification system 120 can generate a print signature that can bederived from the interior regions of the recovered halftone (orstegatone if a payload was encoded for the reference signature). Theprint signature analysis and detection will be described below withrespect to FIGS. 2-9. The forensic recovery and verification system 120compares the recovered print signature to the reference print signaturestored in the registry.

An authentication event occurs by the forensic recovery and verificationsystem 120 based on a comparison between the reference signature fromthe registry and the print signature derived from the captured image.This can include a statistical analysis (e.g., compare print signatureinterior differences to a statistical threshold, where the threshold isa designated level of statistical confidence to grant authentication).The authentication can include other analysis such as a machine learningor artificial neural network analysis where trained classifiers analyzethe respective print signatures in view of the retrieved referencesignatures from the registry. It is noted that the forensic analysis andverification procedures described herein can be applied to any type ofprinted document such as can be provided by printers or other devicessuch as copiers, fax machines, and multi-function print devices.

FIGS. 2-9 demonstrate some examples of forensic verification that can beimplemented utilizing print signatures. Such description will beprovided in terms of specific examples such a scenic image as an examplehalftone image, however, any portion of the image can be employed. Also,the examples include description of stegatones which are a particularcategory of halftones. As noted previously, the halftone itself can beemployed by the halftone interior analysis techniques described herein.

For purposes of simplification of explanation, in the present example,various components of the system 100, such as the encoding system 110and the forensic recovery and verification system 120, are illustratedand described as performing different functions. However, one ofordinary skill in the art will understand and appreciate that thefunctions of the described components can be performed by differentcomponents, and the functionality of several components can be combinedand executed on a single component. The components can be implemented,for example, as computer executable instructions (e.g., software,firmware), hardware (e.g., a CPU, an application specific integratedcircuit), or as a combination of both. In other examples, the componentscould be distributed among remote devices across a network, for example.The executable instructions 110 can be provided as a non-transitorycomputer readable medium having the computer executable instructionsstored thereon.

FIG. 2 illustrates an example of a forensic verification encoding system200 for generating reference signatures from halftone images. It isnoted that FIG. 2 will also be described collectively with respect toFIGS. 2-8 which depict example images for encodings and exampleintermediate processing stages for the respective images. With respectto FIG. 2, a stegatone generator 210 receives a payload 212 and agraphic mule image 214 and generates a stegatone and stegatone decodesupport data. The mule image 214 is the portion of an image where datais encoded into the respective halftone. A printer 220 prints thestegatone and generates a secure hardcopy which is photographed orimaged by capture device 230. A preprocessor 240 processes an image fromthe capture device 230. The preprocessor 240 can operate via at leasttwo methods including a bandpass filter depicted in FIG. 4 and adifference method depicted in FIG. 5.

A subdivide area processor 250 utilizes alternative subdivision methodsdepicted in FIGS. 6-8 for processing the pre-processed image from thepreprocessor 240. An area code generator 254 performs final stageprocessing before a print signature of the captured image is saved in aprint signature registry 260. Dashed lines 270 and 280 indicateadditional processing branches, which may vary depending on thepreprocessing and subdivision techniques employed. For example, line 270may include directly storing the payload 212 in the print signatureregistry 260 to facilitate future forensic verification and indexing ofprint signatures. As another example, the line 280 demonstrates that thestegatone may be passed to the preprocessor 240 and subdivide areaprocessor 250 before further processing by the area code generator 254.

The system 200 enables forensic signatures to be extracted from theinterior of halftones (e.g., steganographic halftones) in combinationwith covert embedding of data payloads. One aspect of this combinationis that the payload data can be used as an index to efficiently storeand retrieve the forensic signature to/from the print signature registry260. The captured hardcopy from the capture device 230 can bepreprocessed in order to optimize forensic discrimination, and theinterior area of the halftone can be subdivided in a number ofalternative ways. Systems for both the encoder (FIG. 2) and decoder(FIG. 9) are provided.

A function of the encoding system 200 is the creation of a securehardcopy document with an embedded payload 212 along with filing itsforensic fingerprint in the registry 260. The stegatone generator 210takes the data payload 212 and input image 214 referred to as a “mule”since it is the vehicle that transports the payload when printed. As anexample, one type of input images that can be utilized (others arepossible) include any type of photo-based or graphic-based images. Animage example is shown at 310 of FIG. 3.

After preprocessing, a reference halftone as shown at 314 of FIG. 3 canbe generated from the mule image 214. The reference halftone can be astandard clustered-dot halftone, as shown at 314. Halftone cells can beclassified in a reference map such as 0-bit, 1-bit, 2-bit, or 3-bit datacarriers. Examples of these cells are depicted at 320 of FIG. 3. It isnoted that 0-bit carriers are referred to as reference cells since theyare unchanged from the original cells and can be used for alignment. Forinstance, cells can be reference cells because they are too large to beshifted or too small to be detected. Cells can also be forced to bereference cells if their unaltered, shape is desired to retain edgedetail or if they are needed to assist alignment. The payload 212 can beencoded by means of a single pixel shift of the halftone clusters, forexample. The data carrying capacity of the image of FIG. 3 is 349 bits,for example, which is many times more than most encryption standardsrequire such as AES, DES, FIPS, and so forth. A 349-bit payload can beencoded as the steganographic halftone or “stegatone” as shown at 324 ofFIG. 3.

Along with producing the respective image, the stegatone generator 210of FIG. 2 outputs stegatone decode support data that can be utilized fordecoding the stegatone, possible regeneration of the stegatone, and/oraid in indexing print signatures from the registry 260. Stegatone decodesupport data can include the mule image 214, reference halftone,reference map, along with shift and selection rules for decoding. Thestegatone can be passed to the printer 220 to create the secure hardcopyof the print signature.

In preparation to locating a forensic “finger print” in the interior ofthe halftone or stegatone in this example, the secure hardcopy can bedigitized via the capture device 230 (e.g., digital camera or scanner).The capture hardware should be of sufficiently high resolution to renderforensic quality detail (e.g., 7200 DPI) such as shown at 330 of FIG. 3.A unique identifying print signature can be extracted by measuring thevariance within subareas of the captured stegatone or halftone. To havethe most differentiating code, the captured image should be preprocessedto amplify what is unique at a microscopic scale. The preprocessor 240of FIG. 2 can perform this task. It first should align the capturedstegatone (or halftone) by making use of the stegatone decode supportdata.

One type of preprocessing is shown in FIG. 4, wherein a capturedstegatone 410 can be bandpass-filtered at 420 via bandpass filter 430.Low frequencies can be removed as these frequencies are common to allprints and provide unwanted signal in the signature, and the highestfrequencies can be removed to reduce imaging noise.

A second preprocessing example is illustrated at FIG. 5. In thisexample, the digital stegatone can be utilized and is indicated by thelines 270 and 280 in FIG. 2. A model of the printer rendering processcan be used to filter the digital stegatone to produce a stegatone modelas shown at 510 of FIG. 5. This image can be subtracted from the alignedcaptured hardcopy at 520 to produce a difference signal image shown at530.

A task of the subdivide area processor 250 in FIG. 2 can be used topartition the preprocessed image into small sections. One exampleapproach is to segment the image into equi-spaced blocks. FIG. 6illustrates an example division into 80.times.80 blocks but otherdivision patterns are possible. Larger sample sizes can correspond tocodes with greater discriminative power, but may require morecomputation and storage.

Another subdivision approach can be to segment the image along halftonecell boundaries. For the example image 310 of FIG. 3, there are23.times.23 halftone cells. This segmentation is shown in FIG. 7. In thecase of stegatones, the reference map identifies carrier cells. Carriercells can have symmetric shapes and provide black-to-white transitions.The number of samples then can be reduced to only measuring carriercells depicted as grey in FIG. 7. It is noted that these cells cancorrespond to the carriers as indicated in the reference map 320 of FIG.3. The cells can be further subdivided into concentric shells as shownat 810 of FIG. 8 to offer a measure of anisotropy where weights areassigned to the shells corresponding to their discriminating value. Ifthe digital stegatone is used as a guide, the carrier cell shiftpositions are known and the concentric shells could be likewise shiftedas is shown at 820 of FIG. 8 for the example case where the shift is tothe right. Other shift positions (e.g., up, down, diagonally, to theleft) are also possible.

When the image is subdivided with areas selected, the area codegenerator 254 of FIG. 2 computes a normalized variance in each area.Means to generate this code include computing the variance of each areaand from them compute a mean variance that acts as a unit value. Thevariance of each area can then be normalized by dividing by the meanunit value. Rounding this normalized variance to produce an ordered listof small integer values (one for each area) completes the area codingscheme. This code can be stored in the print signature registry 260. Thedata payload 212 can be extracted from the stegatone and can be used toprovide a reference into this registry. This can both improve efficiencyof subsequent code comparison and improve robustness as a one-to-onecomparison can be more robust than a one-to-many search for the closestmatch, for example.

FIG. 9 illustrates an example of a forensic verification decoding system900 for analyzing reference signatures and print signatures frominteriors of halftones. A function of the system 900 is to recover apayload from a stegatone (or halftone if stegatone not used), use therecovered payload to index a print signature registry 910 and verifythat the print signature of the captured stegatone is consistent withthe reference print signature version stored in the registry.

Given a secure hardcopy document 920, a capture device 924—similar towhat was used in the capture stage of the encoding system of FIG. 2, isused to create a digitized version of the document. This can be passedto a stegatone recovery system 930, along with the stegatone decodesupport data as generated by the encoding system depicted in FIG. 2. Oneaspect to recovering the payload is alignment of the captured image.This implies that the boundaries of each halftone cell are known. Sincethe digital stegatone will not yet be available, the reference halftone(314 of FIG. 3) can be used to align the captured image. The cell shiftscan then be located and the payload recovered. Due to possibleimperfection in the print-capture process, errors can occur. For thisreason error correction codes can be used. Aspects of the procedure ofinterpreting the payload can be analogous to the procedure of readingdigital information from other physical media, such as CDs, or sometypes of disks since the processes that perform measurements todetermine the represented bits are typically not error-free, the datashould be protected with error correction codes. In the case ofstegatones, data can be represented by tiny shifts in sub-elements(e.g., a subset of pixels) of a printed dot cluster. Scratches or foldscan make interpretation of the shift ambiguous. Even without physicaldegradations of the document, nonlinearities in the printer or capturedevice can distort alignment which in term can result in errors in datarecovery. Error correction coding can utilize redundancy in a strategicmanner to detect and correct such errors.

Recovery of the forensic signature can operate in a symmetrical mannerto that used in the encoding system of FIG. 3. When the digitalstegatone is needed for a preprocessor or subdivide area processor 950,it can be regenerated from the recovered payload via a stegatonegenerator 960. Alternatively, the original stegatone can be fetched froma library (e.g., print signature registry 910) and indexed by all orpart of the payload. When an area code is established via an area codegenerator 970, a forensic verification system 980 can validate itagainst a reference version stored in the print signature registry 910.The reference signature can be efficiently located by using all or partof the payload as an index into the registry 910, for example. Dashedline at 960, 984, and 990 represent alternative example paths dependingon whether a stegatone is employed or a halftone is utilized in theencoding/decoding process.

In view of the foregoing structural and functional features describedabove, an example method will be better appreciated with reference toFIG. 10. While, for purposes of simplicity of explanation, the method isshown and described as executing serially, it is to be understood andappreciated that the method is not limited by the illustrated order, asparts of the method could occur in different orders and/or concurrentlyfrom that shown and described herein. Such a method can be executed by aprocessor and associated equipment, for example.

FIG. 10 illustrates an example method 1000 for performing forensicverification of printed media utilizing halftone to generate and analyzeprint signatures. The method 1000 includes recovering a print signaturefrom the interior of a halftone contained in a captured image at 1010(e.g., via print signature extractor 1110 of FIG. 11). At 1020, themethod 1000 includes comparing the print signature to a referencesignature to determine differences between the print signature and thereference signature (e.g., via comparator 1120 of FIG. 11). At 1030, themethod 1000 includes performing a forensic analysis on the signaturesbased on the comparison to verify the image (e.g., via forensic analyzer1130 of FIG. 11). Other aspects of the method 1000, although not shown,can include encoding payload information into the halftone to form astegatone. This can include performing bandpass filtering or digitalsubtraction on the stegatone to facilitate further image analysis. Thiscan also include subdividing the stegatone to mitigate computing andstorage requirements.

FIG. 11 illustrates an example of a forensic analysis system 1100. Thesystem 1100 includes computer executable instructions 1108 that enable aforensic analysis and verification of printed documents. The system 1100includes a print signature extractor 1110 to extract a print signaturefrom an interior of a halftone of an image of printed media. Acomparator 1120 compares the print signature to a reference signaturestored in a registry to determine differences between the printsignature and the reference signature. A forensic analyzer 1130 performsa forensic-level statistical image analysis on the print signature andthe reference signature based on the comparison to authenticate theprinted media.

What have been described above are examples. It is, of course, notpossible to describe every conceivable combination of components ormethodologies, but one of ordinary skill in the art will recognize thatmany further combinations and permutations are possible. Accordingly,the disclosure is intended to embrace all such alterations,modifications, and variations that fall within the scope of thisapplication, including the appended claims. As used herein, the term“includes” means includes but not limited to, the term “including” meansincluding but not limited to. The term “based on” means based at leastin part on. Additionally, where the disclosure or claims recite “a,”“an,” “a first,” or “another” element, or the equivalent thereof, itshould be interpreted to include one or more than one such element,neither requiring nor excluding two or more such elements.

The invention claimed is:
 1. A non-transitory computer readable mediumcomprising computer executable instructions that when executed cause atleast one processor to: extract an interior of a halftone of an image ofprinted media, wherein at least a portion of the interior of thehalftone is encoded with a print signature; access a reference signaturestored in a registry, wherein the reference signature includes encodedpayload information, wherein the halftone is encoded with a payload toform a mapping between the print signature and the payload information;and verify that the printed media is authentic based on a comparison ofthe print signature and the payload information.
 2. The non-transitorycomputer readable medium of claim 1, and further comprising computerexecutable instructions that when executed cause the at least oneprocessor to: employ the payload information to index the printsignature.
 3. The non-transitory computer readable medium of claim 1,and further comprising computer executable instructions that whenexecuted cause the at least one processor to: encode the halftone withthe payload information to forma stegatone.
 4. The non-transitorycomputer readable medium of claim 3, wherein the halftone is encodedwith a unique payload to form a one-to-one mapping between the printsignature and the payload information.
 5. The non-transitory computerreadable medium of claim 3, wherein the halftone is encoded with acommon payload to form a many-to-one mapping between the print signatureand the payload information.
 6. The non-transitory computer readablemedium of claim 1, and further comprising computer executableinstructions that when executed cause the at least one processor to:preprocess the halftone via a bandpass filter to mitigate low and highfrequencies within a captured image.
 7. The non-transitory computerreadable medium of claim 1, and further comprising computer executableinstructions that when executed cause the at least one processor to:preprocess the halftone via a difference analyzer that subtracts animage from n aligned captured hardcopy to produce a difference signalimage.
 8. The non-transitory computer readable medium of claim 1, andfurther comprising computer executable instructions that when executedcause the at least one processor to: process the halftone via asubdivide area processor that uses an equi-spaced grid to segment thehalftone into smaller processing portions.
 9. The non transitorycomputer readable medium of claim 8, and further comprising computerexecutable instructions that when executed cause the at least oneprocessor to: process the halftone according to concentric regions of animage.
 10. The non-transitory computer readable medium of claim 8, andfurther comprising computer executable instructions that when executedcause the at least one processor to: produce a print signature with anarea code generator based on output from the subdivide area processor.11. The non-transitory computer readable medium of claim 10, and furthercomprising computer executable instructions that when executed cause theat least one processor to: determine a normalized variance for eachprocessing portion generated by the subdivide area processor.
 12. Amethod, comprising: recovering a print signature from an interior of ahalftone of a captured, image of printed media, wherein the halftone isencoded with the print signature; comparing the print signature topayload information encoded in a reference signature stored in aregistry, wherein the halftone is encoded with a payload to form amapping between the print signature and the payload information; andverifying that the printed media is authentic based on the comparison.13. The method of claim 12, and further comprising: performing bandpassfiltering or digital subtraction on a stegatone formed by encoding thehalftone with the payload information; and subdividing the stegatone.14. The method of claim 12, and further comprising: indexing the printsignature using the payload information.
 15. A system comprising: amemory for storing computer executable instructions; and a processingunit for accessing the memory and executing the computer executableinstructions, the computer executable instructions comprising: apreprocessor to generate a filtered signal or a difference signal from acaptured image of printed media; a subdivide area processor to segmentthe filtered signal or the difference signal in preparation of furtherimage processing of the captured image of printed media; an area codegenerator to process segmented output from the subdivide area processorand to recover a print signature from an interior of a halftone that isencoded with payload information; and a verification system toauthenticate the printed media by comparing the recovered printsignature with a reference signature stored in a print signatureregistry, wherein the reference signature includes payload informationthat is used to index the print signature, wherein the halftone isencoded with one of a unique payload to form a one-to-one mappingbetween the print signature and the payload information or a commonpayload to form a many-to-one mapping between the print signature andthe payload information.
 16. The method of claim 12, wherein thehalftone is encoded with a unique payload to form a one-to-one mappingbetween the print signature and the payload information.
 17. The methodof claim 12, wherein the halftone is encoded with a common payload toform a many-to-one mapping between the print signature and the payloadinformation.